Bug Proofing Visual Basic: A Guide to Error Handling and Prevention
(Publisher: John Wiley & Sons, Inc.)
Author(s): Rod Stephens
ISBN: 0471323519
Publication Date: 11/01/98

Hold Code Reviews

After you have stepped through a routine in the debugger, you should hold a code review to discuss the routine with other programmers. The review should include the routine’s author (you), at least one experienced programmer, and at least one inexperienced programmer. The three of you provide very different points of view that allow you to catch more bugs than any one of you would separately.

As the routine’s author, you know how the code works, or at least how it is intended to work. Explaining the code to the others helps make your ideas concrete. Often the process of explaining the code verbally or in writing is enough to help you discover new bugs.

Advanced reviewers ask hard “what if” questions. They can probe the routine for weaknesses and look for special cases that are not handled properly. They should pay special attention to the routine’s assumptions and error handling code.

Inexperienced reviewers ask questions that others will not. The answers to some of these questions seem obvious to you and more experienced reviewers. Sometimes the answers are so obvious that more experienced programmers will overlook the fact that they are wrong. If you explain the code to a beginner while a more experienced programmer listens, the two of them sometimes find bugs that none of you would find alone. At the same time, the beginner learns more about good programming practices and bugs that may arise in the future.

Test Exhaustively

To perform an exhaustive test, you send every possible input into a routine and verify that the result is correct. This kind of test is almost foolproof. If you test every input the routine might ever encounter, the routine cannot fail to produce correct results later. Note that you still need to protect the routine from unforeseen conditions, such as when a crucial file is missing or the system runs out of memory.

Unfortunately, it is almost never possible to test a routine exhaustively. Most routines must be able to handle so many different inputs that testing every possible combination is impossible. For example, suppose a routine is designed to sort a list of 10 numbers with values between 1 and 100. The number of possible arrangements of 10 numbers between 1 and 100 is 100¹⁰ = 10²⁰. Even if you could test 1 million of these combinations per second, it would take more than 3 million years to test them all. Many routines used in real applications have far more possible inputs than this.

In cases when you cannot test exhaustively, you can turn to black box and white box testing.

Perform Black Box Testing

In a black box test, you treat a routine as if it were a black box. You dump inputs into the routine, see what comes out, and verify that the results are correct. You use no information about how the routine works or what goes on inside the box to pick the test cases.

If you can test every possible input, black box testing turns into exhaustive testing. Normally, however, you do not have time to test exhaustively. Instead, you can generate a large number of random inputs, dump them into the black box, and verify the results. If you test several million of the 1020 possible input combinations, you will have some chance of detecting the routine’s most frequently occurring bugs.

Black box tests are usually much easier to run than either exhaustive or white box testing, so you should run them whenever possible.

Perform White Box Testing

When you perform a white box test, you are allowed to peek inside the box and see how the routine works. Using that information, you devise the most treacherous combinations of inputs that you can. You pick the nastiest special cases and the weirdest values to create inputs that are likely to cause problems for the routine. The following list presents some general guidelines you can follow to flush out bugs.

Do the unexpected. Pick inputs that will not generally be expected by the routine. Test unusual and impossible conditions. If the routine takes a person’s age as an input, try passing it the ages 0, –1, and 90,000. If the routine assumes an input is nonzero, try to invoke it with that input set to 0. Make sure the routine verifies its arguments and catches any invalid values.

Follow all paths. Devise inputs that force the routine to execute every line of code. Consider all of the If, Select Case, and other statements that perform decisions. Then try to generate inputs that force every possible combination of those decisions.

Test boundaries. Use inputs that exercise the boundary conditions of arrays, collections, and other data structures used by the routine. If an array has dimensions 1 to 100, see what happens if you try to access entries 0, 1, 100, and 101. Try a few items somewhere in the middle, too.

Do nothing. Try to omit inputs to the routine. If the code takes optional parameters, omit some in every possible combination. If the routine works with a data structure, try passing it one that is empty or that has not been initialized.

In general, discovering the absolutely best possible set of inputs is difficult. The book, The Art of Software Testing, by Glenford Myers (John Wiley & Sons, 1979) provides an in-depth look at selecting the best possible test cases for white box testing. It shows how to design the smallest possible number of test cases that are still likely to find any bugs the routine contains.

Consider Global Variables

When you test a routine, remember that global variables may be part of a routine’s inputs. If the routine uses global variables and data structures, you need to test their effects, too. Be sure to consider both module-global and system-global objects.

Plan Tests

Plan tests in advance and allow sufficient time for them. Make up a chart of all of the different combinations of test technique, scope, and development phase that make sense and plan tests for them.

Write the tests explicitly into the project schedule. If developers see that the tests are expected, they are more likely to perform them.

Follow each project activity immediately with the tests needed to find any bugs introduced by the activity. After you write a routine, test it. After you write a specification, test it. Try to catch the bugs as early as possible. They will only become harder to remove later.

Test Continuously

The previous sections have mentioned this, but it is so important that it is worth repeating: test continuously. If you leave testing to the end of the project, you guarantee unwelcome surprises. Every developer must test at every stage of the project. Frequent testing allows you to catch bugs before they can cause serious damage.

Have the Proper Attitude

Most programmers find testing a dull chore. They rush through it to get on to something more interesting. Many programmers find bug chasing even more repulsive than testing. If that is the case, you should not think of testing as a horrible task that you should avoid as much as possible. Instead, think of it as an easy way to get out of debugging later. Every hour you spend testing may save you several hours debugging. Tests that catch bugs early let you spend more time writing new code in the long run.

Admit to yourself that you must eventually find and fix all of the bugs. Postponing the inevitable will not make the bugs go away. Find the bugs now using well-focused tests, or find them later when they cause unexpected behavior that may be hard to find and correct.